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ABSTRACT 

The need for a revision of the field Reader 
Evaluation Form used throughout the Regional Research Program, as 
well as by other programs in the Office of Education, is documented. 
An empirical development of a revised form was completed though two 
formative evaluation cycles. The revision was then subjected to a 
field test. Utilizing the revised form, field readers evaluated 
proposals which had previously been submitted to the Regional 
Research Program. As a control measure, previous evaluations were 
available. Comparisons with the existing form indicated that the 
revision provided greater objectivity and more comprehensive and 
specific coverage of proposal components. The mean number of comments 
and ratings per proposal obtained from four reviewers was 195 using 
the revision as compared to 48 via the old form. The revision 
provided equal inter-rater reliability. Decisions tended to be more 
critical, and favorable recommendations were more likely to impose 
conditions upon the proposed project. The revision provided more 
adequate bases for ranking proposals, including several alternative 
scales for ranking. A panel review judged the input from the revision 
to be a more useful evaluation for both decision makers and 
appl icant 8 . (Author /DB) 
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The research reported herein was performed pursuant to a contract with 
the Office of Education, U. S. Department of Health, Education, and 
Welfare. Contractors undertaking such projects under Government spon- 
sorship are encouraged to express freely their professional judgment 
in the conduct of the project. Points of view or opinions stated do 
not, therefore, necessarily represent official Office of Education 
position or policy. 
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ABSTRACT 



The need for a revision of the Field Reader Evaluation Form used 
throughout the Regional Research Program, as well as by other pro- 
grams in the Office of Education, is documented. A survey of similar 
proposal evaluation forms used in other federal agencies and founda- 
tions indicated that such forms had limitations similar to those of 
the form used by the Regional Research Program. 

An empirical development of a revised form was completed through 
two formative evaluation cycles. The revision was then subjected to 
a field test. Utilizing the revised form, field readers evaluated 
proposals which had previously been submitted to the Regional Research 
Program. As a control measure, previous evaluations were available. 
These evaluations had been made utilizing the existing form. 

Comparisons with the existing form indicated that the revision 
provided greater objectivity, and more comprehensive and specific 
coverage of proposal components. The mean number of comments and 
ratings per proposal obtained from four reviewers was 195 using the 
revision, as compared to 48 via the old form. 

The revision provided equal inter-rater reliability. Decisions 
tended to be more critical, and favorable recoimnendations were more 
likely to impose conditions upon the proposed project. The revision 
provided more adequate bases for ranking proposals, including several 
alternative scales for ranking. 

A panel review judged the input from the revision to be a more 
useful evaluation for both decision-makers and applicants. 
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PROBLEM 

Background 

This study was supported by a grant from the Regional Research Pro- 
gram of the Office of Education. The purpose of the study was to facilitate 
the proposal evaluation process by developing an improved evaluation form 
for use in reviewing proposals. The study was particularly intended to 
improve the proposal evaluations within the Regional Research Program, but 
also to assist proposal rating throughout the Office of Education. 

The Regional Research Program was established in September 1966, and 
offices were opened in each of the DREW regions across the country. The 
specific goals of the Regional Research Program were: 

1. To support small-scale educational research projects. 

2. To facilitate participation in research over a broad 
range of educational professionals. 

3. To encourage small colleges to undertake research programs 
so that students may benefit from staff who are engaged in 
research activities. 

4. To provide for direct and expeditious handling of proposals. 
The regional program was designea to sim. lify application 
procedures and to be more sensitive to gi^assroots needs. 
The program was particularly aimed at encouraging new 
researchers, those lacking extensive experience. A limit of 
$10,000 was Imposed upon all grants. 

All proposals submitted to the Regional Research Program were 
reviewed by field readers in that particular region. The field readers 
utilized a form supplied by the Bureau of Research of the Office of 
Education. This form was also widely used in other Office of Education 
evaluations of proposals. A copy of the form is contained in Appendix A. 

The Bureau of Research Field Reader Evaluation Form thus played a 
key role in the annual investment of millions of dollars by the Office 
of Education. Through the use of this form, field readers arrived at 
a recommendation to approve or disapprove small project grant applica- 
tions as well as applications for large grants and contracts throughout 
USOE. 
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There was evidence readily available from field readers, directors 
of the Regional Research Program, grant applicants, and third-party 
evaluators that this proposal evaluation form was in serious need of 
revision. Despite substantial evidence of the need for revision, and 
despite the importance of the form in the award of esearch grants and 
contracts, the Field Reader Evaluation Form had not been i«ccorded the 
investment of systematic empirical development revision. The reader 
should be aware that the Regional Research Prog am was terminated shortly 
after this project was initiated. The point is discussed under the 
Procedures section, inasmuch as the old Field Reader Form was also 
used to evaluate a wide range of proposals, and was not limited to 
use within the Regional Research Program, an improvement of the form 
has potential impact beyond that program. 

The cruciality of the evaluation forms is manifest in several 
aspects of the USOE funding program. The forms are the basis for 
the field readers' ratings and evaluation of proposals. In panel re- 
views of proposals, the forms are the basis for the panel discussions. 
The forms are usually the only recorded documentation of field reader 
evaluation and comments relative to each proposal. Feedback received 
by grant applicants as to the quality of their proposals must be de- 
rived from the forms. Thus, they shape the investment of funds and 
provide a basis for closing the feedback loop to the initiators of 

proposed projects. 

The people who completed the forms — the field readers— felt that 
the forms needed revision. The author contacted more than 20 experienced 
field readers in Regions IX and X by letter, telephone, or personal con- 
versation. All reported the need for and usefulness of revising the 
evaluation forms. 

A comprehensive survey, funded by the Office of Education, confirmed 
the need for revision. A 1970 report by the Bureau of Applied Social 
Research, Columbia University, reviewed a number of aspects of the 
Regional Research Program (Rogers, et al., 1970). In this study, 423 
field readers in all regions were polled through a mailed questionnaire. 
Over 75% of the field readers who returned the mailed questionnaires 
suggested various revisions in the evaluation form. Eleven suggestions 
for changing the evaluation form were provided by the Columbia team. 
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Unfortunately, the suggestions froo the study were never implemented, 
and no revised form was developed for field study. These suggestions 
were given consideration in the present study* 

A number of field readers, finding the forms inadequate for sys- 
tematic evaluation of proposals, developed their own forms which they 
used as a first evaluation, and then translated their own evaluations 
onto the Office of Education forma. For example, one experienced field 
reader in Region IX used a 5-point scale for each of 22 separate 
criteria. 

Similarly, the Directors of the Regional Research Program expressed 
the need for revision of the field reader forms. The author contacted 
seven of the ten Regional Directors. Three Regional Directors reported 
that they were actuallv using some modification of the form. Six of 
the seven were emphatic about tht need and importance of revision. This 
wai congruent with the findings of the Columbia study, in which nine 
Regional Directors were interviewed. All were in favor of revising the 
form, although details of specific suggestions were not contained in 
the report. 

Although applicants do not typically use or even see the Field 
Reader Evaluation Fora, reports from applicants also confirm the need 
for revision. In the Columbia study, which queried 665 applicants, 88X 
of the applicants who had requested an explanation of the funding 
decision were not satisfied with the reply. They wanted a relatively 
specific critique of their proposal. Instead, they received general 
cownents and vague or gross judgments. The reasons for the dissatis- 
faction of grant applicants are readily apparent and hinge upon the 
inadequacy of the Field Reader Evaluation Form. 

The Regional Research Offices were not overstaffed. For example, 
the only professional staff in the Regional Research Office was usually 
the director himself. Lack of time and resources prevented him from 
authoring or substantially editing a detailed evaluation o.^ each proposal. 
The comnents on the field reader forms were his only source of feedback to the 
applicant. Although most of the Regional Directors, and an overwhelming 
majority of the field teadets (Rogers, et al. , 1970), felt that field 
reader comments should be given directly to the applicant, the forms did 
not lend themselves to specific and useful feedback. The directors found 
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clarifying and editing comments a time-consuming task; often comments 
were too cryptic or difficult to interpret for the applicant. 

For two years th.^ author participated in the development of training 
materials designed to improve proposal writing skills for professionals 
ia the research and development areas of education. The trainees were 
typically at the same professional experience level as the majority of 
those who made grant applications to the Regional Research Program—i.e. 
assistant professors, doctoral candidates, federal projects coordinators 
and directors of research for school districts, etc. Part of tUe training 
program involved the evaluation of proposals. Adequate feedback to the 
trainees could not be achieved with the Field Reader Evaluation Form. 
More detailed and specific rating scales, as well as provision for comments, 
were required. The approach finally evolved is described in other publica- 
tions (Crawford & Kieismeier, 1970; Kielsmeier. 1970). 

Direction and Extent of Revisions Needed 

The old Bureau of Research Field Reader Evaluation Form has been 
widely used throughout the Cfil-e of Education for a number of years. Its 
use w;-» i,«!vei lirited to th*» Regional Research Program or to the small 
grants and contracts ^ It has been frequently used for th^ evaluation of 
large projects. Through the cooperation of the Director of Regional 
Research Program and the Contracts Officer in Region IX, the author was 
afforded the opportunity of reviewing the completed forms covering several 
years of proposals. Following is a summary of that review. 

Stripped of instructions, identifying headings, etc., the old form 
consists of six major parts. They are: 

1. The final decision of the reviewer, together with the recommended 
disposition of the proposal. 

Four alternatives are allowed: approval, provisional approval, disap- 
proval, and deferral. The reviewer is asked to provide a brief rationale 
for the decision. This item elicits only very brief statements. Perusal 
of over 300 completed forms revealed that most rationales were limited 
to two or three phrases, or at the most three or four sentences. 



2. A five-point rating scale on which the reviewer makes an overall 
rating of the proposal, but only he has recommended ei*-her approval or 
provisional approval. 

Instructions on the ^orm request the reviewer to te as specific as 
possible and use additional sheets if necessary in cc^npleting the next 
four parts* Such exhortations appear to have little effect* The author 
did not locate one instance in which additional sheets were used or 
required. Typically, the comments consisted of only a word or two. 

3. Educational Significance. In parentheses are the following 
questions: Does the proposal address itself to an important educational 
problem? Does the proposal have a sound theoretical basis? Can the 
anticipated results be sufficiently generalized? What is the relation 
to similar known research? 

The apparent attempt is to have the reviewer address each such rele- 
vant aspect of educational significance. Most reviewers, however, seem 
content with an overall sweep at this item. 

4. Personnel and Facilities (Have the principal investigators the 
professional competence and experience to carry out the work of the 
proposal? Does the applicant or applicant organization give assurance 
of the necessary facilities — space, personnel, equipment, etc. — for 
performance of the work?) This section usually elicits very little comment. 
"Okay" or "appears adequate" are typical. 

.5. Research Design (Are the prrblems, objectives, procedures, and 
the relationships among all three clearly and logically stated? Does 
the statement of procedures to be followed include, where applicable, 
information on sampling techniques, controls, data to be gathered, 
instruments to be used, and statistical and other analyses to be made?) 

Every aspect of methodology is lumped under this item. It is up 
to the reviewer how comprehensive and exhaustive his comments will be. 
Typically, reviewers comment on what they feel to be the most salient 
features of the proposal. They make no effort to exhaustively walk 
through objectives, designs, each aspect of the procedures, scheduling, 
analysis, etc. 

6. Economic Efficiency (Is there a favorable relationship between 
the probable outcome of the project and the total effort expenled?) 



This is most frequently answered in a yes/no fashion. Rarely are 
any comments included* 

Only one rating is requii ^ by the form — that for the overall 
proposal* The components are little more than general headings* The 
specificity and comprehensiveness of the comments are up to the reviewer. 
The form does not assist him with any kind of checklist or provide 
assurance that the reviewer attends to each major component of the 
proposal, let alone aspects within components. The form provides little 
in the way of assistance, req rements or restrictions. It is an oper* 
form, open to noise and subjectivity. 

Communication from field readers and Regional Directors pointed to 
the need for the addition of more objective rating scales, provision for 
comments, as well as ratings on more specific aspects of the proposal, 
and the addition of more explicit criteria. 

In the Columbia study 69Z of the field readers suggested separate 
ratings and comments for personnel and facilities. Sixty-two percent 
suggested the inclusion of a rating scale for each item on the form. 
Thirty-six percent suggested the addition of a special perforated addition 
to the form which could be torn off. This perforated section would be 
available for comments directed exclusively to Office of Education personnel 
and not meant to be returned to the applicant. 

The author's own experience and communication with other field 
readers and Regional Directors, in general, confirmed the feeling that 
the objectivity, specificity and usefulness of the form needed to be 
increased. However, the inclusion of a rating scale and checklist for 
specific items did not preclude provision for open-ended comments. The 
expansion and specification of the conglomerate grouped under "Research 
Design" into specific components of methodology appeared to have high 
priority. 

The American Institutes for Research faced a similar problem in its 
Creative Talent Awards for doctoral dissertations. The program has been 
in effect for approximately ten years, and awards are made on a national 
basis to various categories of doctoral research dissertations in the fields 
of psychology and education. The review panel consists of the most dis- 
tinguished figures in psychology and education. A roster of reviewers 
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resembles the list of past presidents of the Ar»ierican Psychological 
Association and the American Educational Research Association. Despite 
this highly selective identification of field readers, evaluations of 
dissertations were found early in the program to be uneven. Many 
evaluations were subject to the same criticism as the Field Reader 
Evaluation Forms, e.g. cryptic, gross, difficult to interpret, etc. 
The American Institutes for Research evolved a nine-point rating scale 
for each of a number of criteria regarding the dissertation. Adequate 
space for conanents on each item was also provided. Although still 
subject to revision, this approach has resulted in considerably more 
specific and more useful evaluations and explanations of the basis of 
the evaluations. An analogous approach is now being extended within 
AIR to all of its final reports. 
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The general objective of the proposed project was to develop a 
more effective field reader evaluation form. The revised form should 
provide more specific, comprehensive and objective evaluation of pro- 
posals than had been provided by the existing Bureau of Research form. 
The form needed to be designed so that it did not present an undue 
burden on scantily paid field readers, but rather facilitated their 
review of proposals* The form also needed to be designed so that 
specific and comprehensive feedback to applicants could be readily 
obtained directly from the form > with minimal editing burden placed 
upon Office of Education personnel. Feedback derived from such a 
revised form should provide a more adequate basis for the revision and 
improvement of proposals than has been feasible with the existing form. 

The revised form should thus provide: 

1. an increased number of specific evaluation comments and 
ratings relevant to proposal components; 

2. a more objective evaluation with provision for open-ended 
explanations and comments; 

3. increased inter-rater reliability across components and 
across total ratings; 
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4. a convenient and reliable basis for ranking proposals; 
5* a more useful evaluation, as judged by directors of 
research programs and by proposal authors. 

By developing a more effective form which would improve proposal 
evaluations this project should yield benefits to several major groups 
in the research community: 

• USOE Research Programs, by providing a higher standard of 
proposal evaluation. 

• Research grant applicants, by providing comprehensive 
critiques that will be more useful in improving the 
quality of subsequent applications. 

• Field readers, by facilitating the application of their 
experience and knowledge to each proposal, and by pro- 
viding a more adequate checklist and format. 

PROCEDURES 

General Approach 

The approach used was an empirically-based product development 
approach. It is the approach frequently used in the systematic develop- 
ment of courses, curricula, and other educational materials. Steps 
cited as essential to the approach vary depending on the degree of detail 
deemed important (see Chapter III, "Instructional System Development*' 
of the CORD National Research Training Manual, Crawford, et . al., 1969). 
However, an overall sequence such as that described for the present pro- 
ject is typical. The major steps include: 

1. development of specifications; 

2. development of measures to assess degree of attainment 
of objectives; 

3. design, tryout, formative evaluation, and revision of 
prototype version; 

4. additional tryout, evaluation, and revision; 
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5. design and conduct a field test of the system; 

6. revision as indicated on the basis of the field test 
evaluation* 

The project was originally designed to coordinate with the proposal 
evaluations of the Regional Research Program. Arrangements had been 
made with several directors to conduct tryouts with their field reader 
panels. However, after the project commenced, the Regional Research 
Program was terminated. The ten regional offices were closed and the 
directors were assigned to other programs. At the present date, the 
regional offices have not been reopened and will not be within the 
immediate future. The program which this project was intended to help 
has been wiped out. This raises the issue of why a form for a non~ 
existent program should be revised. 

An approved proposal evaluation form should have extended use beyond 
a particular program. We should note again that the old Bureau of 
Research form was used not only by the Regional Research Program, but 
also by many other programs. All programs that require the submission 
of proposals need effective proposal evaluation. Full and painstaking 
evaluation of these proposals is a must, whether it is done internally 
by staff members, by independent field readers via mail, or by a panel. 
An evaluation form is a useful facilitator under any of these conditions. 
For example, although panel members meet in a group to discuss the points 
of each proposal and arrive at some consensus, a common basis for start- 
ing the panel discussion is required. The author has observed a number 
of panels in which the initial degree of input for panel members was so 
varied that valuable time was wasted in working up to a useful common 
discussion. A properly designed form can provide a starting framework 
that offers reasonable assurance that each member of the panel has 
attended to the major points of concern within the proposal. 

In addition, the function of the form to serve as feedback to the 
applicant is an important one. No matter what system of evaluation is 
used, the evaluation comments need to be structured so that they can be 
transmitted to the applicant with minimal burden to staff. In the panel 
approach, or any other approach, the comments of each reviewer of the 
\ panel can be clearly expressed on an adequate form. The subsequent task 

-9- 

O 

ERIC 

- - - - - ■ ■ -■■ ^- >^...a,.>..>>^^-^..f...i^^ax:.».^^ 



of the applicant feedback can then be carried primarily by the evalua- 
tion form. Additional conclusions of the panel developed during meet- 
ings may also be added* 

There is a need for an adequate proposal evaluation form in agencies 
other than the Regional Research offices. 

Initial Specifications for the Revision 

Directions for the design of the revision came from several sources. 
Included were: the Columbia Report (Rogers, et al. , 1970); direct 
communication with directors of the Regional Research Program and field 
readers; the experience of the project director in developing proposal 
evaluation procedures for the CORD National Research Training Manual 
(Crawford, et al. , 1969) and The Proposal Writing Manual and Workbook 
(Crawford & Kielsmeier, 1970); and the experience of staff at the 
American Institutes for Research in solving similar evaluation problems. 
Thoughtful suggestions from Dr. Walter Hirsch, Director of Regional 
Research for Region IX, were particularly upeful to the project. 

Initial specifications for the revision included the following: 

1. The form should elicit a comprehensive set of comments 
from the reviewer covering all major components of 
typical proposals. 

2. The form should facilitate specific rather than global 
comments. 

3. Input from the form should be suitable for feedback to 
applicants. There should be minimal or no need for 
editing. 

4. As much as possible, provision should be made for 
objective ratings throughout the form. 

5. Provision must also be made for open-ended comments by 
reviewers. 

6. The form must not impose a substantially greater time 
burden on the reviewer than the present form. 

To obtain a broader range ot input, inquiries were directed to 
35 offices of the federal government and several major foundations 
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Each respondent was asked to provide proposal evaluation forms that 
had been used by his or her agency, or that had been used by other 
agencies. Inquiries were directed to offices of the following federal 
agencies and foundations: 

Department of Agriculture 
Department of the Army 
Department of Commerce 

Department of Health, Education & Welfare 

Social and Rehabilitation Services 

Social Security Administration 

U.S. Office of Education 
Department of Justice 
Department of Labor 
Department of the Navy 
National Institute of Education ^ 
National Institutes of Health 
National Science Foundation 
Department of Transportation 
Veterans Administration 



Carnegie Corporation of New York 
The Ford Motor Company Foundation 
Rockefeller Foundation 
Russell Sage Foundation 

Twenty agencies responded. The majority of those responding 
indicated they did not employ a proposal evaluation form. Several 
federal agencies indicated that they design an evaluation form 
for each set of contract proposals submitted in response to an indi- 
vidual RFP (Request for Proposaxs). The effectiveness and cost 
effectiveness of creating a new form for each solicitation seems open 
to question. 

A total of 11 agencies indicated that they were using proposal 
evaluation forms. Examoles of these forms are included in Appendix B. 
The similarity of these to the Bureau of Research form is close. In 
fact, one of the forms, OE Form 9020, is almost identical except that 
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ratings are requested for each major section. Problems associated 
with the Bureau of Research form, such as lack of objectivity, fail- 
ure to generate specific comments across the major dimensions of 
proposals, etc., would appear to be equally associated with the other 
forms located. However, all forms obtained were reviewed for sugges- 
tions as to format, dimensions, rating procedures, etc. 

Development of Measures to Assess the Degree of Attainment of Objectives 
of the Revision 

Measures were keyed to the objectives identified previously. Measure 
that appeared to be most appropriate include: 

1. A measure of the extent to which use of the form results 
in a comprehensive critique of a proposal. This involved 
counting and a weighted summation of the number of specific 
objective ratings and comments dealing with the major com- 
ponents of a proposal. 

2. Inter-rater reliability with respect to the summative ratings 
of the proposal and the recommendations for funding. 

3. Ranking proposals via the form. The degree to which 
the form provided a i jliable and useful means of rank- 
ing, particularly ranking proposals in the approved 
category. 

4. The usability of the form as measured by time required 
to complete it and its adequacy and appropriateness as 
judged by experienced field readers. 

Design and Tryout of the Initial Prototype 

An initial prototype was designed following the specifications 
described above. The initial selection of the format, extent and 
content of the form required a series of decisions involving trade- 
offs among alternative solutions. A number of alternatives were 
included in the first phases of tryout. 

The form was then given a tryout utilizing a simulated proposal 
review by each of five senior AIR staff members. Following the 
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tryout, each reviewer, working independently* critiqued the form via 
a written evaluation, including an item-by-item review. Following 
this, each reviewer discussed his critique and his suggested revisions 
individually with the project director. 

Input derived from this step was voluminous. Recommendations 
ranged from detailed changes in the wording of an item through a dis- 
cussion of major alternative approaches to the overall problem of 
evaluation. Changes which were congruent with the original objectives, 
and which were supported by a consensus of the panel, were incorpojated 
into the second prototype. Revisions that were conflicting or discrep- 
ant with other attributes of the form or presented potential disadvantages 
with respect to the original objectives were subjected to a series of 
trade-offs. 

A second version of the revised form was readied for a tryout. At 
this time several noticeable features of the revision included: 

1. Provision for rating each item. Every attribute of the 
proposal identified by the form was tied to a rating 
scale. 

2. Provision for reviewers' comments on each item, 
rather than by major sections only. 

3. Provision for the attributes of both developmental and 
research proposals within a single form. 

4. Provision of a separate section for comments addressed 
to the funding agency only. Part of the instructions 
informed reviewers that their other comments could be sent 
directly to applicants but that this section would be 
treated confidentially. 

Additional Tryout, Evaluation! and Revision 

It was originally intended to send the prototype form to the 
directors of the Regional Research Program for a tryout with field 
readers in their area. This procedure was impossible after the 
collapse of the Regional Research Program. As an alternative approach, 
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a list of former readers in Regions IX and X was compiled, and the 
cooperation of a panel of these experienced reviewers was obtained. 
The prototype form was then sent to each of 12 field readers, together 
with a statement of the project objectives. An additional form was 
also prepared on which field readers rated the adequacy of the revision. 
For comparison purposes, a copy of the existing Bureau of Research form 
was attached. Tvelve field readers then rated each item of the revised 
form, and compared the new form with the old one in light of the objec- 
tives of the project. 

The overall results, comparing the revision to the old form, were 
almost unanimously favorable to the revision. More specific inputs 
from this evaluation weir in the direction of simplifying and shorten- 
ing the new form. At this time the revision extended to 12 pages. 
Yhe more global items were sharpened or dim' iated. Several redun- 
dancies and ambiguities were identified and removed. Increased emphasis 
was given to those attributes of proposals which dealt with the statement 
of a problem. This third revision of the form was then readied for a 
field test. 
Field Test 

From tne files of the Regional Research Program in Region IX 
eight previously submitted proposals were selected. Four of these 
were research and four developmental proposals. Within each set of 
fourt two had been approved for funding and two had been approved or 
provisionally approved, but had not been funded. Evaluations by four 
field readers, using the old Bureau of Research form, were available 

for each proposal. 

All identifying names, institutional affiliations, geographic 
locations, etc., were then removed from each proposal which became 
"real" but anonymous. From a list of the former fi<»ld readers in 
Regions IX and X a sample of 16 readers was selected. These readers 
possessed similar qualifications to those used in the previous tryout. 
However, no one who had served in that formative evaluation participated 
in the field test. 

The cooperation of each reader was requested. Fifteen agreed. An 
additional name was selected from the field reader list to substitute 
for the sixteenth, who was too busy to participate. Two proposals. 



together with copies of Che new form, were mailed to each reader. The 
readers were instructed to evaluate the proposals using the new fonr 
in a similar fashion to that which they had followed for the Regional 
Research Program. Each field reader received one research and one 
developmental proposal. By using this design, four independent evalua- 
tions by field readers using the new form were obtained for each proposal. 
As a comparison basis, the four original evaluations from the old form 
were available. Thus, the results of the field test were based on eight 
proposals, each of which had four evaluations via the old form and four 
evaluations via the new form. 

After reviewing the proposals, each field reader also completed an 
evaluation of the new form, using an instrument especially prepared for 
this purpose. Following the return of the materials, each reader received 
an honorarium comparable to that offered under the Regional Research 
Program. 

RESULTS 

The results will be examined in light of the objectives of the 
proj ect. 

Qblective ; The revision will result in an increased number of specific 
evaluation comments and ratings. 

Did the new form yield such an increase? With respect to ratings, 
the total number of ratings derived from the old rorm was 30 for the 32 
evaluations of the eight proposals. This represents a mean of less than 
one rating per proposal by each reviewer. The revised form yielded 891 
ratings, with a mean of 28 ratings per proposal by each reviewer. The 
quantity of ratings is summarized in Table 1. 

TABLE 1 

Number of Ratings by Individual Reviewer 

Range* Mean* Total 

Old Form 0-5 1 30 

New Form 20-31 28 891 



*These figures refer to the number of ratings 
made on a proposal by each reviewer. Thus, 
the mean number of ratings made on a proposal 
by one reviewer using the new form is 28. 

-15- 



As noted, ratings from the old form total 30, somewhat less 
than the total number of reviews, 32. This is due to the instruc- 
tion on the form which directed the reviewer to rate only acceptable 
or provisionally acceptable proposals* IVo reviewers recommended 
against funding, and so did not rate that proposal* 

How specific were the ratings? The one rating yielded by the old 
form was a global rating with respect to the overal adequacy of the 
proposal. The revised form provides for 25 specific ratings. There 
are five specific ratings regarding aspects of the problem; tv,o specific 
ratings regarding objectives; ten regarding procedures; five on 
personnel and facilities; and three on impact and effectiveness. There 
are, in addition, five ratings which relate more globally to proposal 
components. The revised form used in the field test is contained in 
Appendix C. 

The total number of comments obtained from the old form was 310. 
The revision yielded 671. The mean number of comments per proposal 
are respectively 9.7 for the old form and 21.0 for the revision. 

The differences in relative number of ratings and comments elicited 
are illustrated in Figure 1, which presents a comparison for each of the 
eight proposals. 

Statistical tests for mean differences between the old and the 
new forms with respect to comments per proposal, r-itings per proposal, 
and both comments and ratings indicated that those differences exceed 
the .001 level of probability. 

Objective: Increased inter-rater reliability. 

Achievement of this objective should be revealed in increased agree* 
ment among reviewers of a proposal. The two forms differed markedly 
internally; however, both did request a final decision using three similar 
alternatives: approval, provisional approval, and disapproval. Actually, 

he old form provided a fourth alternative — deferral. For the present 
purpose, we may appropriately combine the fourth alternative with disap* 
proval so that both forms present the same alternatives. 

Furthermore, the categories of provisional approval and approval 
both represent favorable recommendations by the reviewer. In the latter 
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case, suggestions are included in the decision or requests for additional 
information are made. Frequently, in the author's experience usiYig the 
old form, proposals with provisional approvals were accompanied by 
as high or higher ratings than approved proposals. 

What was the degree of unanimity among the decisions of the 
four reviewers of each proposal? With the old form, there was 100a 
agreement on final disposition among the reviewers on six of the eight 
proporals. On two proposals, there was 75% agreement, i.e. three of 
the four reviewers agreed. With the revised form seven proposals showed 
100% agreement on final disposition and the remaining one showed 307c 
agreement. The degree of variance of reviewer decisions is shown in 
Table 2. 

TABLE 2 

Reviewer Agreement on Recommended Oisposi . .. 

100% 75% 50;: 

Agreement Agreement Agreeruen t 

Old Form 6 2 U 

Ne.; Form 7 0 1 

The spread of decisions using the revised form is ;k compcjrn t I v t 
tendency to arrive at more critical recoinnendations. Across the ei.;iu 
proposals, two reviewer recommendations for disapproval were given on 
the old form. Using the revision, ten recommendations for disapproval 
were obtained. In addition, recommendations for provisional appro^'al 
rather than approval were more frequent. Reviewers tended to suggest more 
conditions that should be met before funding. 

Objective : Provide a more convenient and reliabl*? basis tor ranking 
proposals. 

The only basis for ranking using the old form was the single 
overall rating. Typically, proposals in the approved category and those 
in the provisionally approved category were ranked, based upon the mean 
overall rating given by the several reviewers. The old form provided 
only this one rating as a basis for ranking. If proposals were tied in 
overall ranking, no objective index was available ro break the ties. 
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The new form, however, provides in addition to the overall rating, 
a rating for each major proposal component. Thus, the reviewer rates 
the proposal in terms of the sections dealing with problems, objectives, 
procedures, personnel and facilities, efficiency/effectiveness. Utiliz- 
ing these component ratings, other indices such as a simple sum or mean 
are readily available in addition to the overall rating. 

Furthermore, the form provides 25 specific ratings other than the 
summative ratings of components, indices are also readily derived from 
these specific ratings, for example, a summed score across the specific 
ratings. Thus, as an initial basis, the new form provides three mea- 
sures, any, all, or combinations of which could form the basis for rank- 
ing; mean overall rating of the proposal; mean ratings of the proposal 
components; and the mean specific ratings. In Table 3 indices are shown 
for the eight proposals used in the field test. It is apparent from 
this that the first three proposals are in a three-way tie if we use 
the mean overall rating derived from the old form. Using the new form, 
this is not the case. Perhaps this is due to the use of a nine-point 
scale for rating. However, the new form also provides adequate infor- 
mation so that proposals may be ranked not only on measures of overall 
rating, but ranked with respect to proposal components and specific 
ratings. Thus, using the revised form, one might rank the eight pro- 
posals pither by mean component ratings, by mean specific ratings, or 
by combinations of these with overall ratings. 



TABLE 3 
Mean Ratings 



Old Form 


New Form 


Proposal 


Mean Overall 
RatliiK 


Mean Overall 
Rating 


Mean 
Component 
Ratine 


Mean 
Specific 
Rating 


R-1 


2 


4.51 


4.75 


4.83 


R-2 


2 


5.22 


5.45 


5.41 


R-3 


2 


4.00 


5.00 


4.95 


R-4 


4 


5.00 


5.23 


5.24 


D-1 


2.50 


4.33 


4.80 


5.48 


D-2 


1.25 


6.75 


5.37 


5.90 


D-3 


1.50 


5.00 


4.80 


4.99 


D-4 


3 


2.50 


2.70' 


3.23 
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Further information is provided by the new form with respect to 
each component of the proposal. Ratings are available separately for 
the five major components of each proposal. In Table 4 summed ratings 
for both specific and overall components are presented for these five 
dimensions of each of the eight proposals. 

In Table 4 the eight proposals are shown together with sufficient 
information to rank them by components. Other indices are readily 
derivable. For example, the overall rating could be summed with the 
sum of the ratings for components. The form lends itself to the develop- 
ment of both simple and more esoteric scales. The proposals can be 
ranked on each component and an overall sum derived based on mean rank 
across components; or a weighting procedure could be added so that one 
or more components, e.g. the significance of the problem, is given 
additional weight. The revision represents a significant advance over 
the old form in making available additional information by which pro- 
posals may be usefully ranked. 

Objective : Provide a more useful evaluation to program directors and 
applicants. 

We had originally intended to obtain evidence regarding the useful- 
ness of the revision through a trial run in the Regional Research Program. 
Feedback from the Regional Directors and proposal applicants would have 
been obtained. However, the untimely demise of that program prevented 
this. 

Six senior research scientists at the Palo Altc office of AIR were 
presented with the old and the revised forms, together with the result- 
ing evaluations on the field test proposals. They were then asked to 
rate the degree to which the revision resulted in a more or less useful 
evaluation. The results were unanimous in identifying the marked 
superiority of the revision. A similar degree of preference was shown 
by junior staff using the same procedures. Junior staff were selected 
whose experience was comparable to that of the graduate student or 
junior-level professor (who were the applicants for whom the Regional 
Research Program was mainly intended). 
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To assist the reader in forming his or her own judgment of the 
usefulness of the form, we have compiled comparable evaluations of 
one proposal* The evaluations of four field readers are combined into 
one form. First the old form is presented, then the new form. (These 
are presented at the end of .his section, beginning on page 23.) 

Several factors in addition to inter-rater reliability bear upon 
the usefulness of the revision: 

1» Does it require an inordinate amount of time to complete? 
The experience of the developers indicated that all staff completed 
evaluations of small grant proposals within one hour using the re\ ised 
form. This appears to be borne out by the field test. Several reviewers 
estimated their time as one hour or less per proposal. Of the 16 re- 
viewers, 13 judged the time required as reasonable. Three felt it 
required too much time, particularly for small grant proposals. 

2. Does the form cover all major ^proposal dimensions? 
Thirteen of the field test reviewers affirmed that it did. Three 

felt some additions should be made. Those additions specifically sug- 
gested were among those the project staff had considered and finally 
rejected in the early prototype development. The suggestions included: 
additional ambiguity-clarity items; separately identiiied process and 
product objectives; and some items on reliability and validity of instru- 
ments and procedures. 

3. Is the form adequate for both research and development proposals? 
We had asked this question continually throughout the project. 

Eleven of the twelve field reader reviewers of the second prototype trial 
judged that version adequate for both kinds of proposals. On the field 
test, twelve of the reviewers judged it adequate. Four felt that it did 
not adequately cover developmental proposals. Suggested changes were: 
more emphasis on evaluation and use of product; and more items on eco- 
nomic efficiency. 

In response to the question of what should be consolidated or 
eliminated, most reviewers (10 of 16) replied that the form was not 
over inclusive or redundant. Specific suggestions for elimination 
tended to be varied and contradictive. One reviewer's redundancy was 
another's meat. Two general suggestions for reducing the length of 
the form were: 
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1. Eliminate the overall ratings within proposal 
components. 

2. Provide one space for comments for each proposal 
component rather than for each item. 

Both of these suggestions had been considered and had appeared on an 
early prototype. The second suggestion does result in a marked reduc* 
tion in paper and pages required for the form. However, both were 
rejected on the grounds that they reduced the useful input from reviewers. 

Evaluation Input from Field Readers 
of Proposal R-1 via the Bureau of Research Form 



A. Educational Significance 

1. Support for the proposed longitudinal study of preschoolers is 
to be highly recommended on several counts: First, it examines 
centrally important education problem areas. Second, the present 
study if based on extensive prior work (testing of 800 3-4 year 
olds). Not to follow up with these children now would be very 
wasteful. Third, the problem is well defined and will be 
directed by a well respected, productive researcher. Fourth, 

the study will contribute to needed developments of mental tests 
for the young preschool and primary grade child. This proposal 
is for a longitudinal step in an important (major) study of the 
development (emergence) of divergent thinking and relationships 
of that development to parental and environmental factors. 

2. Positive on all counts. 

3. The problem is important and relevant. The theoretical basis 
is good. The results will be generalizable. Other related 
research is identified in the proposal. 

4. Excellent — while a theoretical framework is not spelled out in 
this proposal, such a framework does exist. I must admit to 
some disappointment in not finding such material in this 
proposal. 



B. Personnel and Facilities 

1. Staff and facilities most adequate. 

2. Excellent . 

3. Good. 

4. Adequate. 



C. Research Design 



1. No negative comment. 

2. General design Is completely satisfactory. Lacks details of 
specific analyses, but I do not believe thase to be necessary 
for an experienced researcher In this field of endeavor. 

3. All Is adequate — that which Is not specified In the proposal 
one could Infer with relative safety that the Investigator 

has the required competence to provide the necessary expertise. 

4. While I approve this proposal, the following precautions should 
be reemphaslzed: 

(1) The analyses alluded to In #1 of this critique should 
be noted. 

(2) All of the questions and/or hypotheses which this 
research Is supposed to answer and/or test should 
be spelled out very carefully. This Is perhaps the 
most neglected area In this proposal. 



D. Economic Efficiency 

1. Budget appears reasonable. 

2. Very low cost for a study of this nature. 

3. Excellent — It Is a bargain at the price due to already completed 
work. 

4. Okay. 
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Evaluation Input from Proposal R-1 via the Revised Form 



PROBLEM 

1. Significance of the problem to society: 

1 2/ 3 A SV 6 7 8^/ 9 

Minor Moderate High 

Comments (as part of your cooroentSt specify the area within which you are 
rating the significance of the problem): 

A. Problem is especially acute in urban centers. But, has much 
generalizability to suburbs, 

B. Distinction between cognition and productive thinking is important; 
little explored in preschool child. Significance is not communicated 
adequately in this proposal, however. 

C. No comment. 

D. Of little social impact except in the broad academic sense. 



2. Significance of the problem to science (or to the general advancement of 
knowledge) : 

1 2 3 4 5 6v 7i 8// J 

Minor Moderate High 

Comments (as part of your comments, specify the area within which you are 
rating the significance of the problem): 



A. The problem as stated by the investigator would tend to add knowl- 
edge to the developmental worker and to the vast contribution of 
Piaget, et al. 

B. Changes over a 2~year period in cognition, convergent thinking, and 
divergent thinking is significant problem — probably generalizable to 
similar socioeconomic, and cultural groups. Relation of these fac- 
tors to the problem is not clear. 

C. No comment. 

D. The conceptual area — it will be useful to learn of the developmental 
sequences of thinking processes and ability in early ages. Also, 
the relationship between parental activity and children's thinking. 
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3. Clarity of statement of the problem: 

^ 2/ 3 4 5£ 6 8 9 

Inadequate Moderate Excellent 

Coinnents ; 

A. Well stated. Calls for collection of data over longitudinal period. 

B. Purpose seems to be to develop an instrument, which is not stated 
as the problem or the objective. 

C. No comment. 

n. Quite straightforward. 

4. Relationship of the problem to theory: 

1 2/ 3 4 5^^ 6 7 8/ 9 

Not clear or reflects Moderate Clearly 

lack of understanding Integrated 

Comments ; 

A. Selected related research contributed significantly to the problem 
and its definition. 

B. Review of research on cognitive theory is seriously needed; or 
investigator's own definitions of terms. Why are Guilford's other 
•'operations" left out? Is there not some relation of Guilford's 
"content" dimension? "Product"— on what basis is the classification 
of items made (p. 6)? Relationship of Guilford and Piaget? 

C. No comment 

D. Does not get into underlying assumptions to any sufficient extent, 
except to relate the competition for conformity processes with 
creative or divergent processes. 

5. Relationship of the problem to previous research (or development) : 

2/ ^ 4 5^ 6 ll '—r—^' 

Not clear, incoinplete. S^d^^^^ ''''^i^lii'^rateu 

or reflects lack of inregrateu 

undcrstandius 
Comments : 

A. Obviously the investigator has done work in the area rnd should 
be encouraged to complete this facet. 

B. Reflects lack of knowledge of previous research; lack of search. 
Such a review should suggest hypotheses on the "changes" and 
developmental consistencies. I frankly don't believe the "suggestive 
evidence" from the four-year-old study and find no evidence for 
believing. 
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C. No comment. 



D. No comment. 

6. My overall rating of the problem section of this proposal is: 

1 2 V 4 5v 6 7// 8 9 

Inadequate Adequate Excellent 

Comments (Please include your ratings and comments on any important dimen- 
sions of the proposol not covered by the previous items.) 

A. This evaluator was delighted to observe the identification of 
many variables which can affect preliterate children. 

B. Lack of knowledge on cognitive development in age group the 
investigator proposes to study. Related studies are cited, but 
any particular contribution or relevance to present study is 
obscure. References are not consistent with citations so a 
reviewer can't verify the statements. 

C. No comment. 

D. No comment. 

OBJECTIVES 

1. Relationship of the objectives to the stated problem: 

1 2 ^ 3 ^ 5 6 li 8 9 

Obscure, incomplete. Moderate Well integrated 

or reflects lack of and appropriate 

understanding 

Comments : 

A. Objectives not identified as such. Use three questions which are 
inferred as objectives. Yet, an objective points to some clearly 
identified end. 

B. Questions reflect interest in rearing as dependent variable, state- 
ment of problem suggests longitudinal changes in cognitive style. 

C. No comment. 

D. No' comment. 

2. Extent to which the objectives are stated in measurable terms: 

1 ? / 4 5^ 6^ 7 8 9 

I nad t qua t c Ad cqua t c Exc ell en L 

Conuiients: 



A. For a developmental study they are OK. To a strict behaviorist, 
these objectives are not acceptable. 
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B. They could be. I don*t talnk they have been. 

C. No ccunoent. 

D. Problem is that data from parents Is essentially self-report data, 
with the inaccuracies inherent in them. 

My overall rating of the objectives section of the proposal section is: 

1 2/ 3/ 4 5/ 6 7/ 8 9 

Inadequate Adequate Excellent 

Coromcnts (Please include your ratings and comments on any important dimen- 
sion of the proposal not covered by the previous items.) 

A. Would prefer to observe carefully stated series of objectives. 

B. I don*t know what the objectives are. 

C. No comment. 

D. No comment. 



PROCEDURES 

1. The appropriateness of the design of the project to the solution of the 
problem is: 

1 11 l£ 4 5 6 yj 8 9 

Inadequate Adequate Highly appropriate 

Comments : 

A. Most of the relevant variables are identified. But the researcher 
may have already lost valuable data as the subjects grow older! 

B. Difficult to know when hypotheses are not clear. 

C. No comment. 

D. Not clear as to how the sample will be drawn from the estimated 
800 overall. Expected is about 200. What do the remainder look 
like? 

2. The relationship of the procedures to the implementation of the design of 
the project is: 

1 2/ 3 aV 5 6 7^*^ 8 9 

Obscure or inadequate Adequate Excellent 

Comments : 

A. These are congruent. 

B. Relationship between previous and present research is not clear. 
Procedure is not clear. Analysis ot data is not prescribed. 
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C. No cotnment. 



D. The questionnaire data seems to require a very large matrix. 
Provisions for baseline or comparison groups: 

1 2/ 3 4/ 5 6/ 7/ 8 9 

Inadequate Adequate Excellent 

Comments : 

A. There may be a loss of subjects due to time factor. No where did 
investigator iaentify the precise number of subjects, their general 
accessibility, or where they are located. 

B. No provision for considering original selection (mother's education) 
in follow«*up group. 

C. No comment. \ 

D. See item 1. 

Description of population and sample is: 

1 2/ 3 V 5// 6 7 8 9 

Inadequate Adequate Clear & appropriate 

Conmients : 

A. Could be more carefully described. What are nui|ibers, range of ages, 
location, ease of accessibility. This was very generalized. 

IQ seems to be a function of intellectual production. Sample not 
measured for IQ. 

C. No comment. 

D. No comment. 

Appropriateness of the sample selection procedures is: 

1 2/ 3/ 4 / 5/ 6 7 8 9 

Inadequate Adequate Highly appropriate 

Comroents : 

A. Some tendency to place blame on the NIE and those reviewing the 
proposal for the investigator's lack of foresight and planning to 
develop the proposal. Seemed to be an **If you don't hurry and fund 
this, you'll be sorry** implication* 

B. If results are to be generalized to non-whites, it is inadequate. 

C. No comment. 



D. Unclear. 
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Extent to which controls are provided: 

1 IJ 3/ A 5/ 6 IjL 



Inadequate Adequate Excellent 

Comments ; 

A. Basically, previously collected data will be the control data. 
In this sense, #6 is N/A. 

B. Not needed if the study were truly longitudinal or follow-up 

were more complete. One-quarter of original sample is not adequate. 

C. No coonent. 

D. Difficult to retain consistent controls on this type of problem. 
Measures specified to assess the attainment of project objectives: 

1 2/ 3 4/ 5/ 6 Z 11 

Xn^aequate Adequate Excellent 

Comments : 

A. Test construction and methodology well specified. 

B. Home questionnaire is superf icial-^oes not get at nature of 
interaction between parent and child. 

C. No comment. 

D. Essentially correlational. 
Data collection procedures are: 

1 2/ 3 4/ 5/ 6/ 7 8 

Inadequate Adequate Txccllcnt 

Conimc*nts : 

A. The interview schedule is too "loose.** Needs more continua rather 
than absolutes. Yes-no category is not relevant — note requests for 
time with child, talking to him, etc. 

B. Interview is poor. Derivation of test is unclear. 

C. No comment. 

D. Except that it will be necessary to bear in mind that c)ie parental 
data have only self-report validity, the procedures are adequate. 
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9. The appropriateness of the data analysis is: 

1 2/3/4/ 5 6 7 ^ 9 

I^dequate Adequate Highly appropriate 

Comments; 

A. Should yield results. 

B. Difficulty level — how? 
Reliability— how? 
Factor analysis — how? 

C. No coment. 

D. Not particularly clear — reader must infer adequacy. 

10. The extent to which the project schedule is appropriate to the tasks is: 

1 2 3/ / A 51 6/ 7 8 9 

Inadequate Adequate Highly appropriate 

Coromcnls ; 

A. Project schedule very sketchy; not well developed. 

B. No comment. 

C. No comment. 

D. OK* 

11. My overall rating of the procedures section of the proposal is: 

1 2/ 3/ 4 5/ 6/ 7 8 ? 

Inadequate Adequate Excellent 

Comments (Please include your ratings and comments on any important dimen- 
sion of the proposal not covered by the previous items.) 

A. (1) Instruments should be revised to eliminate absolutes; (2) time 
schedule is inadequate; (3) the subjects' whereabouts and avail- 
ability is non-existent. 

B. Investigator appears to lack skill to do this study — unless she 
has skills not used in the writing of the proposal. 

C. No comment. 

D. No comment. 
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PERSONNEL AND FACILITIES 

1. Qualifications of staff to conduct this project: 

1 2/ 3 4 5 6/ 7/ 8/ 9 

Inadequate Moderate Excellent 

Conancnts ; 

A. Seems to be long-time researcher* 

B. See above. 

C. No cooment. 

D. No conment. 

2. Extent to which staff capabilities are related to task requirements: 

I 2/ 3 4 5 6/7/8/9 

lnapproi>riate"lou Moderate Highly appropriate 

junior- or top-ainvy 

Cor.:ri«*nts : 

A. No comment. 

B. Too unsophisticated 

C. No comment* 
D* No comment. 

3. Time commitments to the project from staff: 

1 2 3 4 5 // 6 / 7 / 8 9 

Inadequate -under- Moderate Highly appropriate 

staffed or overstaffed 

Comments : 

A. Seem to be adequate. 

B. No comment. 

C. No comment* 

D. No comment. 

4* Space, equipment, and other facilities for this project: 

1 2 3 4 / 5 // 6 7 8/ 9 

Inadequate Adequate Excellent 

Comments : 

A. Spelled out in proposal. 
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B. No comment. 



C. No comment. 

D. No comment. 

5. Organizational support for the project, including commitments from cooperat- 
ing organizations if necessary: 

1 2 3 4 / 5 // 6 7 8/ S 

Inadequate Adequate Excellent 

Comments ; 

A. OK. 

B. No comments. 
C« No comments. 
D . No commen'.s . 

6. My overall rating of this section of the proposal is: 

1 2 3 / 4/ 5/ 6 7_/ 8 9 

Inadequate Adequate Ex eljcnt 

Comracnts (Please incJude your ratings and comments on any important dimen- 
sion of the proposal not covered by the previous items.) 

A. No comment. 

B. Principal investigator does not seem to know the contemporary field 
of cognitive development or '-.ognitive style, or intellectual 
functioning in preschool children. 

C. No comment. 

D. No comment. 



EFFICIENCY/IMPACT 

1. The relationship between the specified costs of the project and the 
proposed activities is: 

1 2 3 4/5/6 7/ 8/ 9 

Inappropriate Moderately Highly appropriate 

appropriate 

Com m ents : 

A. To complete a longitudinal study for this cost is a great bargain — 
assuming previously done studies we're adequately conducted. 

B. No comment. 

C . No comment . 

D. No comment. 



2. The relationship between the probable outcome of this project, in terms of 
its contribution and the investment required, is: 

1 2/ 3 4«/ 5 6 IJ 8/ 9 

Unfavorable Moderate Highly favorable 

Comments : 



A. Adds to field of child development. Continues what could be a major 
longitudinal study. 

B. No comment. 

C. No comment. 

D. No comment* 

3. Provisions for dissemination/diffusion of the results of the project are: 

1 2^ 3^ 4 6 1^ 8 9 

Inadequate Adequate Excellent 

Comments : 

A. Not well identified. 

B. Vague. 

C. No comment. 



No comment. 



4* My overall rating of this section of the proposal is: 

{ IJ 3 ^ / 5 6 IJJ 8 9 

Inadequate Adequate Excellent 

Comments (Please include your ratings and comments on any important dijnen- 
sion of the proposal not covered by the previous items.) 

A. Considering that all USOE/NIE projects are placed into the ERIC 
system and that professors always publish (or try to), their research 
results, it might be assumed that this section is adequately 
managed . 

B. An exciting area of investigation, but proposal is lacking a suitable 
design. 



C. No comment. 



D. No comment. 



ERIC 
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CONCLUSION 



This revision appears to have achieved its objectives. It pro- 
vides considerably more evaluative input in the way of specific ratings 
and comments than were obtained with the existing form. In addition, 
the revision presents a more objective basis for this input. Decisions 
reached by using the new form display as great a degree of inter-ra'^er 
reliability as before. However, the decisions are more critical and 
are more often accompanied by conditional requirements. The form is 
generally suitable to both research and development proposals. It 
provides an extensive basis for comparative rankings of proposals, and 
offers alternative weighting systems that extend beyond the capability 
of the older form. The revision is usable without extensive guidelines 
and can be completed within modest time requirements. 

For applicants desiring feedback on their proposals, the form 
provides quite specific ratings and comments throughout the major 
aspects of the proposal. 

The revision was designed for the kind of small grant proposal 
submitted to the Regional Research Program. It also appears to be 
ge' ' izable to other categories of proposals. The revision offers 
other agencies a generic form which could be adapted to specific areas 
with a modest additional effort. 
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APPENDIX A 



EXAMPLE OF 

EXISTING BUREAU OF RESEARCH 
FIELD READER EVALUATION FORM 



T 



OE 6017(11-66) Department of Health, Education and Welfare 

Office of Education 
Washington, D. C. 20202 
Bureau of Research 

FIELD READER EVALUATION FORM 



Field Reader 


Date Mailed 


Date Due 


Project Officer 


Date Returned 




Telephone 


Proposal No* 


Amount 



INSTRUCTIONS—This form is furnished for your use in evaluating the attached pro- 
posal. We would appreciate receiving your completed evaluation in the enclosed 
envelope within three (3) weeks unless otherwise indicated. If you cannot review 
the proposal within that time, return it unevaluated so that it may be sent to 
another Reader for review. Since the Field Reader Evaluation is one of the 
principal factors in determining final action on a proposal, we request that you 
be as thorough and explicit as possible in your responses. Please fill in pages 
1 to 4 and return the entire form to us. If you have any questions concerning 
the proposal, feel free to telephone the Project Officer at the number listed 
above. Please return the proposal along with this evaluation form. 

SUMMARY SHEET 

1. Place an "X" in the bracket which best represents your recommendation concern- 
the disposition of the enclosed proposal, and provide a brief rationale below 
the recommendation. (Rationale: include a statement supporting your recommen- 
dation and an elaboration of suggested modifications if you indicated "Provi- 
sior Approval") 

[] Approval — Proposal worthy of support as proposed 

[] Provisional Approval — Proposal worthy of support with negotiable modifi- 
cations 

[] Disapproval — Proposal not worthy of support 

[] Deferral — Proposal unable to be evaluated without additional information 




2. If you gave "Approval" or "Provisional Approval" to the proposal, circle the 
figure below which most accurately defines your over-all rating: 

(High) 1 2 3 4 5 (Low) 
-38- 



You have been selected to review this proposal as one of a group of field readers 
because of your particular competence In the field. Please write a detailed 
analysis of the proposal as It relates to your area of specialization. Continue 
on a separate sheet If necessary 
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For each of thp four criteria listed below, please indicate in what ways the 
proposal does or does not »eet acceptable standards. Be as specific as PO"i- 
ble in your connents, since they will be used in discussing the proposal with 
the initiator (without reference to the source). Attach additional sheets if 
necessary. 

A. Educational Significance (Does the proposal address itself to an important 
educational problem? Does the proposal have a sound theoretical basis? tan 
the anticipated results be sufficiently generalized? VJhat is the relation 
to similar known research?) 



Personnel and Facilities (Have the principal investigators the professional 
competence and experience to carry out the work of the proposal? Does the 
applicant or applicant organization give assurance of the necessary facilities- 
space, personnel, equipr.ent, etc.— for performance of the work?) 



Page 3 
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C. Research Design (Are the problems, objectives, procedures, and the relation- 
ships among all three clearly and logically stated? Does the statement of 
procedures to be followed include, where applicable, information on sampling 
techniques, controls, data to be gathered, instruments to be used, and statis* 
tical and other analyses to be made?) 



D, Economic Efficiency (Is there a favorable relationship between the probable 
outcome of the project and the total effort expended?) 



ERIC 
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APPENDIX B 



EXAMPLES OF 

PROPOSAL EVALUATION FOMIS USED BY VARIOUS AGENCIES 

B-1: OE Form 9020. This is a current form ^ised by 
the Office of Education and is quite similar 
to the existing Bureau of Research form de- 
scribed in this study. 

B-2 through B-10; Examples of forms currently in 
use. Agency markings have been removed. 
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B-1 



DEPARTMENT OF HEALTH. EDUCATION. AND WELFARE 
OFFICE OF COUCATION 
WASHINGTON. O.C 20202 

TECHNICAL AND/OR PROFESSIONAL SERVICES CONTRACTOR 



FliLO READER 


DATE MAILED 


DATE DUE 


PROJECT OFFICER 


DATE RETURNED (OEUSBONLYf 




TELEPHONE (vt^codt, mtmbtr, m 




PROPOSAL NUMtER 


AMOUNT 








$ 



This form is furnished for your use in evaluating the attached proposal. We would appreciate receiving your 
completed evaluation in the enclosed envelope by the due date noted above. If you cannot review the proposal within that 
time» please return it immediately. Please Till in pages I to 4 and return the entire form to us; be as thorough and explicit as 
possible in your responses. If you have any questions concerning the proposal, feel free to contact the Project Officer, 

SUMMARY SHEET 

THESE MAJOR SECTIONS IN THE PROPOSAL EVALUATION BY CIRCLING A NUMBER ON THE SCALE BELOW 
DISCUSS THESE RATINGS IN DETAIL UNDER ITEMS 3A. B. C. AND 0. oclutt. 



EDUCATIONAL SIGNIFICANCE: 
PROJECT OESION: 
PERSONNEL AND PACILITIES: 
ECONOMIC EFFICIENCY: 



LOW 
1 
t 
1 
t 



HIGH 
6 
S 
5 
6 



^b!t?lA1^ 151'''"^ ^"'C" 8^5T REPRESENTS YOUR RECOMMENDATION CONCERNING THE DISPOSITION OF THE 
ENCLOSED PROPOSAL. AND PROVIDE A BRIEF RATIONALE BELOW THE RECOMMENDATION, mJtionsts:i^^^ 
iuppontnf your rwcomtmndttion. If you indxsm ^Trovision^l Appro^^t" or ''Def^rrw/r specify prowtsiont or information n^tdnit 

□ APPROVAL - PropoMi worthy of s upport at propoMd. No tifnifkont modificMiofii noodad. 

□ provisional APPROVAL - PropoMi it bMteolly worthy of Mpport wtth ipoetfk modfficotions indieotod. 

□ deferral - Additioiwl Informttk^ is nt n iwary for propoMi tvaluotion. Tho formulation of a roeommMdation it 

not po t i l h k without furthor ioformation. 

□ disapproval > This application ihould not baaupportad in itt praaMit form. 



SIGNATURE OF f lELO READER 



OE FORM 9020. tO/7t 



DATE 



Pd9« 1 PREVIOUS EDITIONS OF THIS FORM ARE OBSOLfcTE 



ERLC 
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B-1 



mJH THE INITIATOR ««TMOOT REFERENCE TO THE SOURCE. ATTACH ADDITIONAL SHEETS IP NECESSARY 

A. tooCATiONAL tiONinCAHCf fOoM M* pnpomi »0i 9 i»mh aMnm Imlfni* Hrporwtt ««M tdycatioMl l**^*^]^*;^^**!^, ^ 



B. PROIiCT OttlON (Af th0 pfobhnm. obhcthm, procodum. ^ wptewd productL ^nd th0 fofsHomhipi smon§ 9II four eh^rfy snd logk^ly 
mttd? Dom th$ mmnmt of proeoditm to b9 fo$l<^ kKlud9 oM wplicobh An th9 crimion m^munt mH>roprM fot 

qimdom mkwd mid on§fy$m u$$d? Am piopomd tkmlirm rmtktk?) 



OE FOMM •020. 10/71 ^ 2 

ERIC 



T 



B>1 

9& rtHSONNf L AND FACILITIES (Hm9 Ji9 pHfKfp^t Mvmt t fttoa th9 pmhniomi c o »nptitr>f •a^mimf to tmy oyf the ivof* of propotst? 
0009 mo UppMewir omtmKO of tho nmoomrt' f^cmHot - moeo. ponom^. oquipmont, tutfoct sa m^ ond othor oopportho o4mitti$trottto $onrfce$ 
for pof^ormmtcm of tho woHt? Am mom odtqoom mmbon of pofwoomi ood on thor ouffnod offkhntir to oecompffoh tfio ot^octHm of tM profit?) 



90. ECONOMIC EFFICIENCY (It thom 9 fovofobh rolstionship botwoofi tht probobh outcome of tho proioet ond tt>9 totol fundi onpernhd or betwoon i 
proioemd oetMthi ood totof fundi? Pfoom noto sny budfit ifrm thot om quoitionoblo in tormi of boinf oithor unmolUtkatlY htfh or iowj 
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B-1 



4. OTNin PtimN«IT IWyOWMTlOW OW AOOmOMAt COMMCWn. \r AMY 



Er|c ^ :i6^ 



Oi MUM 10^1 



B-2 

GRANT PROPOSAL RATING SHEET 
RESEARCH PROJECT 

PROPOSAL NO.: 91- TITLE: 
PRINCIPAL INVESTIGATOR: 
PROPOSED BY: 

Recommendation of Panel Member 

£j Approve /~ Approve Provisionally (Indicate changes needed in Comments) 

IJ Defer Decision £j Disappi^ove/Resubmit iJ Disapprove 
Enter explanations pertinent to these ratings in Comments below: 
COMMENTS : 



Signature of Panel Member ^ Date 
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UNSOUQTED PROPOSAL EVALUATION FORM 

(Please Read Instructions on Reverse Before Completing) 



I. SUBMITTER: 



2. rHOPOSAL NUMBER: 



3. DATE RECEIVED- 



4. rORWARDED FOR EVALUATION TO 



8^_OATE 

e. NUM0ER OF COPIES 



7., TITLE 



a. PROPOSAL SHOULD ALSO BE EVALUATED BY: 



B. TECHNICAL FACTORS: 


EXCELL. 


GOOD 


FAIR 


POOR 


OTHER 


A. COMPLETENESS OF PROPOSAL 












B. SOUNDNESS OF APPROACH 












C. ADEQUACY OF FACILITIES 












D. ABILITY OF INVESTIGATORS 












E. VALUE AS A RESEARCH PROJECT 












F.. REALISM: LEVEL OF-EFFORT VS 
PROPOSED ACCOMi LIBHMENTS 












6. REALISM: LEVEL^-EFFORT VS 
PROPOSED COSTS 













ID. WHAT CONTACT HAS YOUR OFFICE HAD WITH SUBMITTER9 WHAT INFORMATION HAS HE BEEN OtVENT IF PROPOSAu 
IS CARRY-ON TO EXISTING OR PREVIOUS CONTRACT. IDENTIFY IT. 



II. FUNDING Q 13 Q IS NOT PLANNED BY THIS OFFICE. 



12. ESTIMATED DATE OF FUNDING 



13. EVALUATOR'S COMMENTS: 



AME OF EVALUATOR 



erJc 



19. CONCURRENCE 



le. DATE 
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GUIDE FOR THE PREPARATICM OF SIMIARY STATEMENTS 

Use the headings listed belov; do not substitute "evaluation'* for CRITIQUE. 
Every sunmary statement should include a RESUME^ a DESCRIPTION » and a CRITIQUE. 
Three sections may be sufficient for disapproved applications, but all of the 
headings should be used in other sunmary statements. Complete sentences should 
be used throughout; the budget, for example, should not be described as singly 
'"Reasonable" or "Adequate." Do not use jargon, vernacular expressions, and 
undefined abbreviations in reporting the scientific review. Colloquial 
language used in the reviewers' comoients should be presented in a literary 
style acceptable for scientific reports. 

RESDflL Briefly summarize the proposed project and the essential reasons for 
the recommendation. 

DESCRIPTION . Present a concise description of the proposed project, including 
the aims, procedures, and background. 

CRITIQUE. Discuss the strengths and weaknesses of the proposed project. Are 
the aims logical? Is the approach valid and adequate? Are the proposed 
procedures feasible and appropriate' Will the research produce new data and 
concepts or confirm existing hypotheses? What is the significance and perti- 
nence of the proposed study in relation to the state of the field and the 
import<ince of the aims? For continuation and supplemental requests, comment 
on past progress. The CRITIQUE must reflect the priority score. 

INVESTIGATORS , Discuss the competence and background of the investigators. 
Since only the name of the principal investigator appears routinely on the 
sunmary statement, names should be used in the discussion of other personnel. 

RESOURCES AND ENVIRONMENT. Discuss the facilities, the equipment, and, when 
appropriate, the extent of departmental and interdepartmental cooperation. 
Conment on the availability of any unusual resources, such as special animal 
species, tissue preparations, or clinical case material. 

BUDGET. Is the requested budget realistic? Are all of the requested items 
justified? Itemize and provide specific reasons for reductions in time or 
amount recomnended. For supplementary requests, comment on the supplementary 
budget in relation to the approved parent budget. 

OTHER HEADINGS: 

HISTORY . An involved explanation or description of the administrative 
background should be included as a special section after the RESUME. 

CRITIQUE (MINORITY). Include a minority report when two or more members 
disagree with the recommendation. CRITIQUE (MINORITY) should follow the 
CRITIQUE (MAJORITY). 

HAZARDOUS PROCEDURE. (k>ament on the nature of potentially hazardous 
materials in relation to the proposal and resolution by Study Section. 



B-4 
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EXECUTIVE SECaiETARY'S NOTE, Call attention to possible policy questions 
or other administrative aspects of the application. 

ADDITIONAL CONSIDERATICHIS: 

Collateral Opinions. Summarize when pertinent and influential to the 
reconmendation. Do not mention the reviewer's name or quote him directly. 

Deferral for Project Site Visit or for Additional Information . Indicate 
the reasons for the deferral and how the Study Section used the information 
obtained in reaching its recommendation. This should logically conclude 
the CRITIQUE. 

DRG - Rev. January 1972 



ERLC 
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Guidelines for Preparing Outside Opinions 

The following general areas should be covered in your critique. Naturally 
these ere general areas and do not in any way preclude you from including 
evaluative comments in other areas. 

Uniqueness and Significance : How significant is the proposed 
project in terms of local, regional, or national import? Has this or 
related work been undertaken before and/or at another site? If so, where 
and how does the current project differ or how is it similar? 

2. Design : Is the design of the project adequate to its objective? 
Cite design weaknesses and strengths uncovered along with illustrative 
examples. Tnis item is obviously very important and sufficient attention 
should be directed to an evaluation of the design to assure a thorough 
review of deficiencies and/or strengths. 

3. Evaluation: An evaluation plan is an important component in most 
projects, although certain developmental projects do not lend themselves 
to a formal evaluation. However, all demonstration projects require an 
evaluation plan; a demonstration project without an evaluation is a service 
project and not eligible for support. A detailed review of the evaluation 
plan should be made citing specific instances where the plan is non-existent, 
weak, fragmented or directed towards the wrong target. 

^- Resources and Collaborative Arrangements : Projects often 
require for their success tne collaboration ot a number of organizational 
units and individuals other than the principal investigator and his 
immediate staff. Comment on this aspect of the proposal. 

5. Principal Investigator and Other Project Personnel : Do the 
project personnel indicate an adequate familiarity with relevant 
literature? Does the principal investigator appear competent to direct 
the project? Who are the major assc^ciates of the principal investigator 
and what are their strengths and weaknesses? 

6. Budget : Evaluate major items in the budget and determine 
appropriateness of their aims to the project. Make specific comments 
as to items in the budget. 

7. Other 
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Category Nuihber 


^ / 

^§ / 

ti o / 

« / 
w / 

a /I. 

/ 43 O 
/ <0 43 

/ 5 ^ 
/ 1-4 flj 

/ CO Gk« 

/ ? 


Understanding of the 
Problem 


Technical Adequacy of 
Proposed Approach 


Suitability of Proposed 
Schedule 


n 
o 

o O 
a. 
cn o 
c ;4 
o cu 

43 CO 

a u 
cc 


Conpany Facilities 
Relevant for Project 


TOTAL NUJERICAL RATING 
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GUIDELINES FOR R^VIEVTCRS IN THE ?Rf.?A.^-\T10N OF CRITIQUES 



1. Read the application carefully. This is Impprtant. Reviews based 
on careless reading are unfair r.o the applicant and/or to competing 
applications. They also reflect badly on the reviewer and on the 
Study Section. 

# 

2. If you have rny questions which may reasonably be asked of the 
applicant and without the answers to which you cannot prepare an 
adequate review, report them promptly to the Executive Secretary to 
allow tirne for him to relay the questions to the applicant and to 
distribute the reply to the reviewers. 

3. Principal components of reviews : Primary and Secondary reviewers' 

critiques are the bases tor much of the discussion of applications at 

study section meetings. They also are the priT.e source of data used by 

the Executive Secretary in preparing Sum^arv Sheets. Other sources 

include the applicaiion itself, the Study Section discussion, and site 

reports. You can I'-cilitate the review process by following this outline 

(« 

a. Descriptior : If you are the primary reviewer, you must write a 
description of the proposed project. In this section you shouid 
provide your understanding, without evaluative cQTr.ment or ton e, of 
the objectives, hypotheses, and methods of the project. It should 
not include description of the investigator, resources and environment 
or budget, except where these are essential to the main description. 
They can be described more fully in the appropriate sections which 
follow. 

b. Uniqueness and Significance : Comment on the significance of 

the proposed project in terms of local, regional, or national import. 
Has this or related work been undertaken Defore and/or at another 
site? If so, where - and how does the current project differ or 
how is it similar? 

c. Design : Is the design of the project adequate to its objective? 
Cite design weaknesses and strengths uncovered along with illustra- 
tive examples. This item is obviously very important and sufficient 
attention should be directed to an evaluation of the design to assure 
a thorough review of deficiencies and/or strengths. 

d. Evaluation : An evaluation plan is an important component in most 
projects, although certain developmental projects do not lend them- 
selves to a formal evaluation. However, all demonstration projects 
require an evaluation plan; a demonstration project without an 
evaluation is a service project and not eligible for support. A 
detailed review of the evaluation plan should be made citing specific 
instances where the plan is non-existent, weak, fragmented or 
directed towards the wrong target. 



-53- 



T 



B-7 

2. 



e* Resources and Collaborative Arrangements : Projects often require 
for their success the collaboration of a number of organizational 
units and individuals other than the principal investigator and his 
immediate staff. Comment on this 'asp act of the proposal. 

f. Principal Investigator and other Project Personnel : Do the 
project personnel indicate an adequate familiarity with relevant 
literature? Does the principal investigator appear competent to 
direct the project? Who are the major associates of the principal 
investigator and what are their strengths and weaknesses? 

g» Budget : Evaluate major items in the budget and determine 
appropriateness of their aims to the project • Make specific 
comments as to items in the budget. 

h. Other 

i. Recommendation . You ma> recommend approval in time and amount > 
approval in reduced or increased time and/or amount, deferral for 
further information (which usually but not necessarily indicates a 
site visit), or outright disapproval. When recommending approval in 
reduced or increased amount, please indicate precisely where the 
changes are to be applied and how large they are to be in each 
category, and indicate the rationale if it is not already evident 
from your foregoing comments. Reductions should be related to 
particular budget items which are excessive or to aspects of the 
project which the study section feels should be deleted or altered. 
The obverse applies to increases* 

You may also recommend shifts in budget allocations. 

Reviewers customarily refrain from presenting their recommendations 
until all written reviews have been presented to the study section. 
Therefore, please refrain from stating a recommendation until it is 
called for by the chairman. 

This concludes the outline of a typical review. It corresponds 
closely to the format of the Summary Statements prepared by the 
Executive Secretary after study section action has been completed. 

One Final Note : It would be helpful to the Executive Secretary if 
you would instruct your typist to double-space. 
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FORM FOR USE BY THE REVIEW PANEL IN SCORING COIffEliTIVE GRANT 

RESEARCH PRO?CC// > 



Submitted for Research on 



Institution _ 
Project Titli^ 



Score * 



1. Relevance to present and future research needs. 

2. Quality of proposal. 

3. Competence of principal investigator (s) 

4. Amount of time and attention principal 

investigatorCs) will devote to the project. 

5. Adequacy of facilities and equipment. 

6. Quality of the research program in the 
institution in the relevant area. — 

7. Feasibility of attaining the objectives. 

8. Relationship to on-going and other 
proposed research in the scientific community 

broadly. 

♦Rating Scores: 3 - Superior; 2 - Sfltis factory; 1 - Fair; 0 - Poor 
please do not score those criteria (1-8) for which you do not have 
sufficient infoimacion; simply insert a dash to indicate it was not 
overlooked. 



Overall adjective rating or proposal: Outstanding 


r 


/ 


Outstanding with 


1 


1 


revision 


Acceptable 


1 


/ 


Unacceptable 


1 


1 



If revision is rcconincnded, how should proposal be revised: 



Concnents : 
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PROPOSAL RATING SHEET 

Reviewer PropoMl No.: 

Invetttgator: 

Institution: 

Please return to: 

If possible by: 

Comments {Continue on additional sheet if necessary) 



OVERALL RATING Signature of Reviewer: 

□ excellent 

□ VERY Gooo Other suggested reviewers {optional) : 

□ gooo 

□ fair 
□poor 
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Evaluation of Policy Related Research 



Title - Proposal No. 

Research Area = 

A. Evaluation Design— Comments on innovative ideas, grasp of the problem, originality, etc.. 
as well as misconceptions, oversimplifications, incorrect formulation, adequacy of models, 
etc. 

1 . Design for Assessing Internal Validity— methods proposed to assess internal validity are 
appropriate and cost-effective. Comments: 

a. Facts . 



b. Opinions 



2. Design for Assessing External Validity— methods proposed to assess external validity 
are appropriate and cost-effective. Comments: 



a. Facts 



b. Opinions 



1 
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3. Methods for Establishing Policy Utility— criteria for judging policy relevance of 
particular studies and of sets of related research bearing on given policy instruments. 
Comments: 

. a. Facts 



. b. Opinions 



4. Knowledge of Policy-Related Research Literature 
Comments: 

a. Facts 



b. Opinions 



Evaluation Design Overall Ai^erage Rating 



Dissemination and Utilization— Comments on innovative ideas, grasp of the problem, 
originality, etc.. as well as misconceptions, oversimplifications, incorrect formulation, 
adequacy of models, etc. 

1 > Proposed format of written report ^understandable to funders and to policy makers at 
the federal level, and local level, and by interested citizen groups. 
Comments: 

a. Facts 



b. Opinions 



2 
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dl^n^^?^^^ dissemination and utilization plans-Is user community 

Smm^^^^^^ suggested strategy result in use of the research by that community^ 



Comments 
a. Facts 



b. Opinions 



Dissemination and Utilization Overall Average Rating 

C. Management Plan 

^^Z^^ 1^^.^!:.^. Competence-Researchers^ experience in evaluation 

l??anollm?^^ necessary for conduct of project available in house If not. have 
arrangements been made to obtain them outside'^ 
Comments. 

a. Facts 



b. Opinions 



^i^^lrZn^T^ ^'''^"^"ce m managing similar activities; the pro.ect 

mlnVnlT T.°'''' ""^ °' ''^^ '^^'^ members: support providSl bv 

3 Th *° '^""^^^^ °' consultants. NSF adv^e and°^dVaS 

and/or other personnel and resources at the key decision points or other appCiate 



Comments, 
a. Facts. 



b. Opinions 



B-9 

3. Financial Management— Does the proposer have a system of reflecting actual costs 
incurred and for controlling costs? (To be evaluated by GCO and not reviewers.) 

Overall Average Rating 
Management Plan 

SUMMARY 

Evaluation Design 

Utilization Plan 

Management Plan 

Overall Rating of Proposal 

D. Comments not to be made available to principal investigator. 



Ranking 



4 
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SUGGESTED GUIDELINE FOR REVIEWER'S EVALUATION 

Reviewer ; 
Review Date ; 

Control Number r 
Title of Proposal ; 
Organization; 
Principal Investigator ; 

1. Validity and scientific merit of basic objective and rationale: 

2. Strengths of proposal; 

3. Weaknesses of proposal: 

4. Adequacy of personnel to the proposes research: 

5. Adequacy of facilities: 

6. Is equipment requested appropriate: 

7. Recommendation; 1) Support as proposed; 2) No support; 3) Support 
in part as follows: 

8. Opinion as to the "relevance" of this proposal and/or the length of 
time required to realize a definite contribution 

9* Other comments; 
10. Rate on basis "Scientific Merit": please give a score of 1-5 (1 bei 
highest) . 
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APPENDIX C 



EXAMPLE OF 
REVISED FIELD READER EVALUATION FORM 
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Department of Health , Education, and Welfare 
Office of Education 
Washington, D. C. 20202 

Bureau of Research 



FIELD READER EVALUATION FORM 


Field Reader 


Date Mailed 


Date Due 


Project Officer 


Date Returned 




Telephone 


Proposal No. 


Amount 






$ 



INSTRUCTIONS 

This form is furnished for your use in evaluating the attached proposal, 
l^e would appreciate receiving your completed evaluation in the enclosed envelope 
by the above due date, unless otherwise indicated. If you cannot review the 
proposal within that time, return it unevaluated so that it may be sent to 
another Reader for review. 

Since the Field Reader Evaluation is one of the principal factors in deter- 
mining final action on a proposal, we request that you be as thorough and explicit 
as possible in your responses. Please complete each of the items, including, 
ratings and comments . Make your comments as specific as possible. 

The items have been selected to be applicable to a wide range of proposals. 
If you do consider an item inapplicable, mark it N/A, with appropriate comments. 

In addition to assisting agency evaluation of the proposal, your ratings 
and comments will also be used to p*.ovide feedback to the proposal applicant. 
However^ the identity of Field Readers will not be revealed to applicants. 

The final page of the fora consists of a summary sheet providing for your 
overall rating of the proposal and your recommendations for its disposition. 
An additional space on tne summary sheet is provided for any comments to USOE 
exclusively . These comments will not be sent to applicants. 

If you have any questions concerning the proposal, feel free to telephone 
the Project Officer at the number listed above. 
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PROBLEM 



1* Significance of the problem to society: 

1 2 3 A 5 6 7 8 9 

Minor Moderate High 

Conments (as part of your comnents* specify the area within which you are 
rating the significance of the problem): 



2« Significance of the problem to science (or to the general advancement of 
knowledge) : 

1 2 3 4 5 6 Z § i 

Minor Moderate High 

Comments (as part of your comments, specify the area within which you are 
rating the significance of the problem): 



3. Clarity of stataent of the problem: 

1 2 3 4 5 6 7 8 

Inadequate Moderate Excellent 

Comments: 



Relationship of the problem to theory: 

1 2 3 4 5 6 7 8 9 

Mot clear or reflects Moderate Clearly 

lack of understanding Integrated 

Comaencs : 



ERLC 
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5* Relationship of the problem to previous research (or development): 

1 2^^ 3 4 5 6 7 8 9 

Not clear, incomplete. Moderate Clearly stated and 

or reflects lack of well integrated 

understanding 

Comments: 



6. My overall rating of the problem section of this proposal is: 

1 2 3 4 5 6 7 8 9 

Inadequate Adequate Excellent 

Comments (Please include your ratings and comments on any important dimen- 
sions of the proposal not covered b-^ the previous items.) 



OBJECTIVES 

1. Relationship of the objectives to the stated problem: 

1 2 3 4 5 6 7 8 9 

Obscure, incomplete. Moderate Well integrated 

or reflects lack of and appropriate 

understanding 

Comments: 



2. Extent to which the objectives are stated in measurable terms: 

1 2 3 4 5 6 7 8 

Inadequate Adequate Excellent 

Comments: 
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3. My overall rating of the objectives section of the proposal Is: 

1 2 3 4 5 6 7 8 S 

Inadequate Adequate Excellent 

Comments (Please Include your ratings and comments on any important dimen- 
sion of the proposal not covered by the previous items.) 



PROCEDURES 

1. The appropriateness of the design of the project to the solution of the 
problem is: 

1 2 3 4 5 6 7 8 9 

Inadequate Adequate Highly appropriate 

Comments: 



2. The relationship of the procedures to the Implementation of the design of 
the project is: 

1 2 3 4 5 6 7 8 9 

Obscure or inadequate Adequate Excellent 

Comments: 



3. Provisions fcr baseline or comparison groups: 

I 2 3 4 5 6 7 8 9 



Ad equa to 



Inadequate 
Comments: 



Excellent 



Description of population and sample is: 

1 2 3 4 5 6 Z 8 9 

Inadequate Adequate Clear & appropriate 

Conments; 



Appropriateness of the sample selection procedures is: 

1 2 3 4 5 6 7 8 9 

Inadequate Adequate Highly appropriate 

Comments: 



Extent to which controls are provided: 
1 2 3 4 L 



Inadequate Adequate Excellent 

Comments: 



Measures specified to assess the attainment of project objectives: 

1 2 3 4 5 6 7 8 9 

Inadequate Adequate Excellent 

Comments: 



Data collection procedures are: 

1 2 3 4 5 6 7 8 

Inadequate Adequate Excellent 

Comments: 



67- 



9. The approrpiateness of the data analysis is: 

1 2 3 4 5 6 Z 8 9 

Inadequate Adequate Highly appropriate 



Comments: 



10. The extent to which the project schedule Is appropriate to the tasks Is: 

1 2 3 4 5 6 7 8 9 

Inadequate Adequate Highly appropriate 

Comments: 



11. My overall rating of the procedures section of the proposal Is: 

1 2 3 4 5 6 7 8 9 

Inadequate Adequate Excellent 

Comments (Please Include your ratings and comments on any Important dimen- 
sion of the proposal not covered by the previous Items.) 



PERSONNEL AND FACILITIES 

1. Qualifications of staff to conduct this project: 

1 2 3 4 5 6 Z § I 

Inadequate Moderate Excellent 

Comments: 



2. Extent to which staff capabilities are related to task requirements: 

1 2 3 4 5 6 7 8 9 

Inappropriate—too Moderate Highly appropriate 

junior- or top-heavy 

Conmients: 
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3. Time commitments to the project from staff: 

2 3 A 5 6 7 8 9 

Inadequate-under- Moderate Highly appropria J 

staffed or overstaffed 

Comments; 



4. Space, equipment, and other facilities for this project: 

1 2 3 A 5 6 7 8 

Inadequate Adequate Excellent 

Comments: 



5. Organizational support for the project, including commitments from cooperat- 
ing organizations if necessary: 

1 2 3 A 5 6 7 8 9 

Inad equa te Ad equa t e Exc el lent 

Conmients; 



6. My overall rating of this section of the proposal is: 

1 2 3 A 5 6 7 8 9 

Inadequate Adequate Ex ellent 

Comments (Please include your ratings and comments on any important dimen- 
sion of the proposal not covered by the previous items.) 



EFFICIENCY/ IMPACT 

1. The relationship between the specified costs of the project and the 
proposed activities is: 



Inappropriate Moderately Highly appropriate 

appropriate 

Comments: 



-69- 



The relationship between the probable outcome of this project, in terms of 

its contribution and the investment required, is: 

, 2 3 4 6 7 i 9 

Unfavorable Moderate SiihTT favorable 



Comnents: 



Provisions for dissemination/diffusion of the results of the project are: 

L 5 6 7 8 9 



Inadequate 
Comments: 



Adequate Excellent 



My overall rating of this section of the proposal is: 

1 2 ^4 -^67 8 9 

' inadequate Adequate S^Ziu^ 

Comments (Please include your ratings and comments on any important dimen- 
sion of the proposal not covered by the previous items.) 
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SUMMARY SHEET 



Overall rating of this proposal: 



1 



2 



3 



4 



5 



6 



7 



8 



9 



Inadequate 



Adequate 



Excellent 



My recommendation for the disposition of this proposal is: 
[ ] Approval — worthy of support as proposed. 

[ ] Provisional approval — worthy of support with the following 
negotiable modifications (specify the modifications needed): 



[ ] Disapproval — not worthy of support 

The major factors in this proposal contributing to my overall rating and 
recommended disposition were (briefly identify the factors or refer to 
previous items): 



Comments for funding agency only . (These comments will be treated as confidential 
and will not be made available to proposal applicants.) 



Date: 



Signature of Field Reader 
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Proposal No. 




